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1.  Productivity  measures,  including  numerical  counts  for  the  following  aspects  of 
the  research  project  during  this  reporting  period. 


Refereed  papers  submitted  but  not  yet  published:  1  (accepted) 


Refereed  papers  published:  0 
Unrefereed  reports  and  articles:  3 

Books  or  parts  thereof  submitted  but  not  yet  published:  0 

Books  or  parts  thereof  published:  0 

Patents  filed  but  not  yet  granted:  0 

Patents  granted  (including  software  copyrights):  0 
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Invited  presentations:  2 
Contributed  presentations:  1 
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Honors  received  (fellowships,  technical  society  appointments,  i\\"\  j 
conference  committee  roles,  editorships,  etc.):  3  i  | 

1.  Co-chair,  SPIE  Conference  on  Intelligent  Robots  and 
Computer  Vision  X:  Algorithms  and  Techniques  (Boston, 
November  1991) 
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2.  Program  Committee,  IEEE  Workshop  on  CAD-based  Vision 
(Maui,  June  1991) 


3.  Program  Committee,  SPDE  Conference  on  Applications  of 
Artificial  Intelligence  X:  Machine  Vision  and  Robotics 
(Orlando,  21-23  April  1992) 

Prizes  or  awards  received  (Nobel,  Japan,  Turing,  etc.):  0 
Promotions  obtained:  0 

Graduate  students  supported  >=  25%  of  full  time:  0 

Post-doctoral  researchers  supported  >=  25%  of  full  time:  0 

Minorities  supported  (including  Blacks,  Hispanics,  American  Indians 
and  other  native  Americans  such  as  Aleuts,  Pacific  Islanders,  etc.; 
not  including  Asians  or  Asian- Americans):  0 


1 


cn 

cn 


0) 


CJ 


S  o 
z  o 
coo 

O  LO 

O  I 

0) 

— I  C  — I 
<U  -H  CN 
4-)  r-l  CNI 

r-J  CN 

OJ  £ 


OJ  H 


Ml 

C 


CO 


Vj  Vj 
Q  < 


O 

OO 


Principal  Investigator  Name:  Dr.  Prasanna  Mulgaonkar 

PI  Institution:  SRI  International 

PI  Phone  Number:  (415)  859-6530 

PI  E-mail  Address:  prasanna@erg.sri.com 

Grant  or  Contract  Title:  Data  Driven  Spatial  Reasoning 

Grant  or  Contract  Number:  N00014-90-C-0063 

Reporting  Period:  1  October  1990—30  September  1991 

2.  Detailed  summary  of  technical  progress. 

Our  research  is  aimed  at  the  development  of  computational  techniques  for  hypoth¬ 
esizing  the  shapes  of  hidden  portions  of  unknown  objects  within  a  pile  of  such  objects, 
using  a  dense  range  image  of  the  pile. 

The  techniques  that  we  have  developed  employ  symmetry,  stability,  viewpoint 
independence,  and  object  impenetrability  to  hypothesize  the  unknown  shape  and  dimen¬ 
sions  of  each  visible  object.  The  process  constructs  alternative  hypotheses,  which 
differ  in  the  way  the  visible  portions  of  objects  are  extended  into  the  occluded  regions 
within  the  scene.  To  ensure  that  each  interpretation  is  consistent  with  the  observed 
range  data,  the  known  geometry  of  the  range  sensor  is  used  in  forming  the  hypotheses. 
The  final  result  is  one  or  more  hypothesized  object  configurations,  each  of  which  is 
consistent  with  both  the  sensed  range  data  and  the  physical  constraints  between  objects 
in  contact. 

For  each  resulting  hypothesis,  a  free-body  analysis  is  performed  to  determine  if 
the  hypothesized  configuration  is  stable.  The  hypothesis  with  the  highest  stability 
rating  is  chosen  as  the  most  likely  correct  interpretation. 

Most  prior  work  analyzing  three-dimensional  images  of  objects  has  concentrated 
on  object  recognition.  Model-based  techniques  have  concentrated  on  approaches  to 
represent  and  recognize  instances  of  known  objects.  Other  work  has  concentrated  on 
techniques  for  description  of  surface  shape.  Shape  descriptions,  from  multiple  range 
images  covering  all  sides  of  an  unknown  object,  have  been  used  to  develop  geometric 
models  or  maps. 

Our  earlier  work  on  interpretation  using  generic  shape  models,  extension,  and 
evaluation  has  been  described  elsewhere  and  has  been  used  by  others  as  a  basis  for 
applying  Dempster-Shafer  theory  to  shape  recognition.  In  the  1990-1991  reporting 
period,  we  have  extended  our  previous  work  to  cover  a  complete  exploration  of  possi¬ 
ble  configuration  hypotheses  and  their  evaluation  using  free-body  analysis  in  the  pres¬ 
ence  of  friction.  This  integration  of  vision  and  mechanics  has  not  been  significantly 
explored  by  other  researchers. 

We  begin  with  three  basic  pieces  of  knowledge:  a  range  image  of  a  scene,  the 
geometry  of  the  sensor  used  to  acquire  the  image,  and  the  expectation  that  the  scene 
consists  of  objects  whose  geometries  can  be  represented  by  a  set  of  generic  shapes 
(e.g.,  boxes  and  cylinders). 
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The  range  image  contains  the  three-dimensional  location  of  visible  points  on  the 
surfaces  of  unknown  objects  in  a  pile.  These  surfaces  occlude  the  volumes  of  the 
objects  within  the  scene;  these  are  the  hidden  volumes  we  wish  to  explore. 

We  expect  our  scene  to  consist  of  objects  whose  shapes  are  familiar  but  whose 
dimensions  are  unknown.  In  our  work,  we  restrict  these  shapes  to  prismatic  solids  and 
cylinders  whose  dimensions  are  unknown. 

Given  the  information  about  the  image,  the  sensor,  and  the  expected  shapes,  we 
look  for  surface  patches  consistent  with  our  expected  shapes.  From  these  primitive 
shapes,  we  form  a  hypotheses  about  the  configuration  of  the  objects  in  the  scene.  Due 
to  occlusion,  we  only  have  incomplete  information  about  the  object  extents,  and  there¬ 
fore  multiple  hypotheses  are  possible.  We  use  a  free-body  analysis  technique  for 
evaluation  of  each  hypothesis. 

Each  final  hypothesis  for  the  configuration  of  the  pile  accounts  for  all  the  visible 
data.  Thus,  from  an  image-processing  standpoint,  there  is  no  way  to  discriminate 
among  these  alternative  configurations.  We  therefore  turn  to  a  nonvisual  approach, 
using  the  physical  consequences  (stability)  of  each  proposed  configuration  to  rule  out 
impossible  arrangements  and  to  rank  the  remaining  plausible  hypotheses. 

We  analyze  the  balance  of  forces  and  torques  within  each  hypothesized  configura¬ 
tion  of  the  pile.  Treating  each  object  within  the  hypothesized  configuration  of  the  pile 
as  a  free  body,  this  process  evaluates  all  the  gravitational  and  contact  forces  acting  on 
it.  Following  a  technique  described  by  Fahlman,  we  attempt  to  find  a  static  solution, 
i.e.,  the  assignment  of  plausible  forces  to  each  contact  point  such  that  the  vector  sum 
of  all  forces  and  the  vector  sum  of  all  torques  on  the  object  each  totals  zero. 

This  free-body  analysis  procedure  partitions  the  objects  of  an  hypothesis  into  two 
classes:  balanced  and  unbalanced.  For  the  hypothesis  corresponding  to  the  true 
configuration  of  a  pile,  we  expect  all  objects  to  be  in  the  balanced  class. 

We  establish  a  stability  rating  for  each  hypothesis  as  the  number  of  balanced 
objects  divided  by  the  total  number  of  objects.  Thus,  a  completely  balanced  hypothesis 
has  a  rating  of  1.0,  while  all  others  are  assigned  lower  values. 

The  techniques  described  above  for  the  generation  and  evaluation  of  alternate 
hypotheses  for  the  configuration  of  a  pile  of  objects  have  proven  useful.  They  predict 
the  size  and  shape  of  the  hidden  portions  of  genetically  shaped  objects  in  a  scene. 

They  also  estimate  the  force  relationships  among  the  objects. 

These  techniques,  like  the  hidden  regions  they  hypothesize,  seem  to  be  relatively 
unexplored  by  image  understanding  research.  One  immediate  possibility  for  further 
research  is  the  inclusion  of  a  larger  set  of  generic  shapes.  This  would  not  only  make 
the  process  more  robust  but  would  also  help  define  its  limits. 
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In  the  longer  term,  the  free-body  analysis  outlined  above  could  be  pushed  forward 
in  the  image  understanding  process.  It  could  be  used  to  suggest  the  appropriateness  of 
one  extension  over  another  in  order  to  reduce  the  explosive  number  of  alternative 
hypotheses.  Beyond  that,  it  might  be  used  to  suggest  the  existence  of  an  entirely 
unseen  object  underneath  the  pile. 
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3.  Publications,  presentations,  and  reports. 

Mulgaonkar,  P.G.,  “Scene  Description:  An  Iterative  Approach,”  Proc.  SPIE  Sympo¬ 
sium  on  Intelligent  Robots  and  Computer  Vision  IX:  Algorithms  and  Techniques,  Boston, 
Massachusetts,  4-9  November  1990. 

Mulgaonkar,  P.G.,  Cowan,  C.K.,  and  DeCurtins,  J.,  “Understanding  Object  Config¬ 
urations  Using  Range  Images,”  Accepted  for  publication  in  the  IEEE  Trans.  Pattern 
Analysis  and  Machine  Intelligence,  Special  Issue  on  3D  Computer  Vision,  to  be  published  in 
late  1991. 

DeCurtins,  J.  and  Mulgaonkar,  P.G.,  “Scene  Description:  Interactive  Computation 
of  Stability  with  Friction,”  Invited  paper,  SPEE  Symposium  on  Intelligent  Robots  and 
Computer  Vision  IX:  Algorithms  and  Techniques,  Boston,  Massachusetts,  November 
1991. 

Mulgaonkar,  P.G.,  Cowan,  C.K.,  and  DeCurtins,  J.,  “Understanding  Object  Config¬ 
urations  Using  Range  Images,”  Submitted  to  the  IEEE  Conference  on  Computer  Vision 
and  Pattern  Recognition,  1992. 

Mulgaonkar,  P.G.,  Cowan,  C.,  and  DeCurtins,  J.,  “A  System  to  Understand  Object 
Configurations  Using  Range  Images,”  SRI  International  Technical  Note, 
ITAD-8842-TN-91-6,  June  1991. 
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4.  Transitions  and  DoD  interactions. 

None  have  taken  place. 
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5.  Software  and  hardware  prototypes. 

SRI  has  developed  a  Sun-based  software  system,  written  in  C,  for  interpreting 
dense  range  images.  The  system  is  portable  and  can  be  modified  to  process  range 
images  from  different  sources.  The  results  are  displayed  graphically  on  a  color 
monitor  for  evaluation. 
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